This script takes the raw outputs from the water quality sensors at Falling Creek and combines them into one file, identifies gaps in the data, runs the data through the QAQC function (qaqc_fcr), and produces QAQC plots for visual inspection.
Use this file path in Carey Lab Reservoir GitHub Repo: “./Data/DataAlreadyUploadedToEDI/EDIProductionFiles/MakeEML_FCRcatwalk/2023/”
This identifies if there are any daily data gaps in the long-term record
This identifies if there are any sub-daily gaps in the long-term record. Most of these gaps happen when we change the program on the data logger.
## [1] "2018-07-04 13:50:00 UTC" "2018-07-05 13:50:00 UTC"
## [1] "2018-08-06 09:00:00 UTC" "2018-08-06 10:00:00 UTC"
## [1] "2018-08-06 10:00:00 UTC" "2018-08-06 10:10:00 UTC"
## [1] "2018-08-06 10:10:00 UTC" "2018-08-06 10:30:00 UTC"
## [1] "2018-08-06 10:30:00 UTC" "2018-08-06 11:10:00 UTC"
## [1] "2018-08-06 11:10:00 UTC" "2018-08-06 11:40:00 UTC"
## [1] "2018-08-06 11:40:00 UTC" "2018-08-06 12:00:00 UTC"
## [1] "2018-08-27 12:10:00 UTC" "2018-08-27 16:50:00 UTC"
## [1] "2018-08-28 11:00:00 UTC" "2018-08-28 14:50:00 UTC"
## [1] "2019-04-15 09:40:00 UTC" "2019-04-15 10:10:00 UTC"
## [1] "2019-04-15 10:10:00 UTC" "2019-04-15 10:30:00 UTC"
## [1] "2019-04-15 10:30:00 UTC" "2019-04-15 10:50:00 UTC"
## [1] "2019-04-15 10:50:00 UTC" "2019-04-15 11:10:00 UTC"
## [1] "2019-04-15 11:10:00 UTC" "2019-04-15 11:30:00 UTC"
## [1] "2019-05-20 12:20:00 UTC" "2019-05-20 12:30:00 UTC"
## [1] "2020-05-15 15:00:00 UTC" "2020-05-15 15:10:00 UTC"
## [1] "2020-10-09 08:00:00 UTC" "2020-10-09 09:20:00 UTC"
## [1] "2020-10-19 14:00:00 UTC" "2020-10-19 17:00:00 UTC"
## [1] "2020-10-20 10:00:00 UTC" "2020-10-20 10:40:00 UTC"
## [1] "2020-10-20 11:00:00 UTC" "2020-10-20 11:10:00 UTC"
## [1] "2020-10-20 12:00:00 UTC" "2020-10-20 12:10:00 UTC"
## [1] "2020-10-20 13:00:00 UTC" "2020-10-20 13:40:00 UTC"
## [1] "2020-10-22 09:00:00 UTC" "2020-10-22 09:30:00 UTC"
## [1] "2021-01-07 10:20:00 UTC" "2021-01-07 10:30:00 UTC"
## [1] "2021-01-07 11:00:00 UTC" "2021-01-07 11:30:00 UTC"
## [1] "2021-02-26 11:00:00 UTC" "2021-02-26 11:02:00 UTC"
## [1] "2021-02-26 12:00:00 UTC" "2021-02-26 12:40:00 UTC"
## [1] "2021-06-28 13:00:00 UTC" "2021-06-28 14:00:00 UTC"
## [1] "2021-08-17 06:00:00 UTC" "2021-08-17 06:10:00 UTC"
## [1] "2022-05-02 13:20:00 UTC" "2022-05-02 13:30:00 UTC"
## [1] "2022-05-05 08:30:00 UTC" "2022-05-05 08:50:00 UTC"
## [1] "2023-01-24 14:00:00 UTC" "2023-01-24 14:10:00 UTC"
## [1] "2023-02-03 13:50:00 UTC" "2023-02-03 14:10:00 UTC"
## [1] "2023-02-07 11:50:00 UTC" "2023-02-07 12:00:00 UTC"
## [1] "2023-03-07 12:00:00 UTC" "2023-03-07 12:20:00 UTC"
## [1] "2023-04-18 11:40:00 UTC" "2023-04-18 13:30:00 UTC"
## [1] "2023-05-29 07:20:00 UTC" "2023-05-29 11:00:00 UTC"
## [1] "2023-06-26 11:10:00 UTC" "2023-06-26 12:10:00 UTC"
## [1] "2023-07-19 09:20:00 UTC" "2023-07-19 10:20:00 UTC"
## [1] "2023-07-31 12:50:00 UTC" "2023-07-31 14:40:00 UTC"
## [1] "2023-08-07 12:20:00 UTC" "2023-08-07 12:30:00 UTC"
This section reads in the QAQC file and then you can look at the head, tail and structure. Make sure the last row is Dec. 31 23:50 of the publishing year. There should be 83 columns unless a new one has been added.
## Rows: 272972 Columns: 83
## ── Column specification ────────────────────────────────────────────────────────
## Delimiter: ","
## chr (1): Reservoir
## dbl (81): Site, ThermistorTemp_C_surface, ThermistorTemp_C_1, ThermistorTem...
## dttm (1): DateTime
##
## ℹ Use `spec()` to retrieve the full column specification for this data.
## ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.
## # A tibble: 6 × 83
## Reservoir Site DateTime ThermistorTemp_C_surf…¹ ThermistorTemp_C_1
## <chr> <dbl> <dttm> <dbl> <dbl>
## 1 FCR 50 2018-07-05 13:50:00 29.0 27.9
## 2 FCR 50 2018-07-05 14:00:00 29.2 28.0
## 3 FCR 50 2018-07-05 14:10:00 29.3 28.1
## 4 FCR 50 2018-07-05 14:20:00 30.1 28.1
## 5 FCR 50 2018-07-05 14:30:00 29.8 28.2
## 6 FCR 50 2018-07-05 14:40:00 29.6 28.3
## # ℹ abbreviated name: ¹ThermistorTemp_C_surface
## # ℹ 78 more variables: ThermistorTemp_C_2 <dbl>, ThermistorTemp_C_3 <dbl>,
## # ThermistorTemp_C_4 <dbl>, ThermistorTemp_C_5 <dbl>,
## # ThermistorTemp_C_6 <dbl>, ThermistorTemp_C_7 <dbl>,
## # ThermistorTemp_C_8 <dbl>, ThermistorTemp_C_9 <dbl>, RDO_mgL_5 <dbl>,
## # RDOsat_percent_5 <dbl>, RDO_mgL_5_adjusted <dbl>,
## # RDOsat_percent_5_adjusted <dbl>, RDOTemp_C_5 <dbl>, RDO_mgL_9 <dbl>, …
## # A tibble: 6 × 83
## Reservoir Site DateTime ThermistorTemp_C_surf…¹ ThermistorTemp_C_1
## <chr> <dbl> <dttm> <dbl> <dbl>
## 1 FCR 50 2023-09-18 23:50:00 21.6 21.6
## 2 FCR 50 2023-09-19 00:00:00 21.6 21.6
## 3 FCR 50 2023-09-19 00:10:00 21.6 21.5
## 4 FCR 50 2023-09-19 00:20:00 21.5 21.5
## 5 FCR 50 2023-09-19 00:30:00 21.5 21.5
## 6 FCR 50 2023-09-19 00:40:00 21.5 21.5
## # ℹ abbreviated name: ¹ThermistorTemp_C_surface
## # ℹ 78 more variables: ThermistorTemp_C_2 <dbl>, ThermistorTemp_C_3 <dbl>,
## # ThermistorTemp_C_4 <dbl>, ThermistorTemp_C_5 <dbl>,
## # ThermistorTemp_C_6 <dbl>, ThermistorTemp_C_7 <dbl>,
## # ThermistorTemp_C_8 <dbl>, ThermistorTemp_C_9 <dbl>, RDO_mgL_5 <dbl>,
## # RDOsat_percent_5 <dbl>, RDO_mgL_5_adjusted <dbl>,
## # RDOsat_percent_5_adjusted <dbl>, RDOTemp_C_5 <dbl>, RDO_mgL_9 <dbl>, …
## tibble [272,972 × 83] (S3: tbl_df/tbl/data.frame)
## $ Reservoir : chr [1:272972] "FCR" "FCR" "FCR" "FCR" ...
## $ Site : num [1:272972] 50 50 50 50 50 50 50 50 50 50 ...
## $ DateTime : POSIXct[1:272972], format: "2018-07-05 13:50:00" "2018-07-05 14:00:00" ...
## $ ThermistorTemp_C_surface : num [1:272972] 29 29.2 29.3 30.1 29.8 ...
## $ ThermistorTemp_C_1 : num [1:272972] 27.9 28 28.1 28.1 28.2 ...
## $ ThermistorTemp_C_2 : num [1:272972] 24.6 24.5 24.3 24.4 24.5 ...
## $ ThermistorTemp_C_3 : num [1:272972] 17.6 17.7 17.8 17.9 17.8 ...
## $ ThermistorTemp_C_4 : num [1:272972] 13.6 13.6 13.8 13.9 14 ...
## $ ThermistorTemp_C_5 : num [1:272972] 11.2 11.2 11.2 11.2 11.2 ...
## $ ThermistorTemp_C_6 : num [1:272972] 11.2 11.2 11.2 11.2 11.2 ...
## $ ThermistorTemp_C_7 : num [1:272972] 11.2 11.2 11.2 11.2 11.2 ...
## $ ThermistorTemp_C_8 : num [1:272972] 11.2 11.2 11.2 11.2 11.2 ...
## $ ThermistorTemp_C_9 : num [1:272972] 11.1 11.1 11.1 11.1 11.1 ...
## $ RDO_mgL_5 : num [1:272972] 7.61 7.58 7.91 7.87 7.71 ...
## $ RDOsat_percent_5 : num [1:272972] 69.4 69.1 72.1 71.8 70.3 ...
## $ RDO_mgL_5_adjusted : num [1:272972] 7.61 7.58 7.91 7.87 7.71 ...
## $ RDOsat_percent_5_adjusted : num [1:272972] 69.4 69.1 72.1 71.8 70.3 ...
## $ RDOTemp_C_5 : num [1:272972] 11.2 11.2 11.2 11.2 11.2 ...
## $ RDO_mgL_9 : num [1:272972] NA NA NA NA NA NA NA NA NA NA ...
## $ RDOsat_percent_9 : num [1:272972] 70.7 70.6 70.7 70.7 70.7 ...
## $ RDO_mgL_9_adjusted : num [1:272972] NA NA NA NA NA NA NA NA NA NA ...
## $ RDOsat_percent_9_adjusted : num [1:272972] 70.7 70.6 70.7 70.7 70.7 ...
## $ RDOTemp_C_9 : num [1:272972] 11.2 11.2 11.2 11.2 11.2 ...
## $ EXOTemp_C_1 : num [1:272972] NA NA NA NA NA NA NA NA NA NA ...
## $ EXOCond_uScm_1 : num [1:272972] NA NA NA NA NA NA NA NA NA NA ...
## $ EXOSpCond_uScm_1 : num [1:272972] NA NA NA NA NA NA NA NA NA NA ...
## $ EXOTDS_mgL_1 : num [1:272972] NA NA NA NA NA NA NA NA NA NA ...
## $ EXODOsat_percent_1 : num [1:272972] NA NA NA NA NA NA NA NA NA NA ...
## $ EXODO_mgL_1 : num [1:272972] NA NA NA NA NA NA NA NA NA NA ...
## $ EXOChla_RFU_1 : num [1:272972] NA NA NA NA NA NA NA NA NA NA ...
## $ EXOChla_ugL_1 : num [1:272972] NA NA NA NA NA NA NA NA NA NA ...
## $ EXOBGAPC_RFU_1 : num [1:272972] NA NA NA NA NA NA NA NA NA NA ...
## $ EXOBGAPC_ugL_1 : num [1:272972] NA NA NA NA NA NA NA NA NA NA ...
## $ EXOfDOM_RFU_1 : num [1:272972] NA NA NA NA NA NA NA NA NA NA ...
## $ EXOfDOM_QSU_1 : num [1:272972] NA NA NA NA NA NA NA NA NA NA ...
## $ EXOTurbidity_FNU_1 : num [1:272972] NA NA NA NA NA NA NA NA NA NA ...
## $ EXOPressure_psi : num [1:272972] NA NA NA NA NA NA NA NA NA NA ...
## $ EXODepth_m : num [1:272972] NA NA NA NA NA NA NA NA NA NA ...
## $ EXOBattery_V : num [1:272972] NA NA NA NA NA NA NA NA NA NA ...
## $ EXOCablepower_V : num [1:272972] NA NA NA NA NA NA NA NA NA NA ...
## $ EXOWiper_V : num [1:272972] NA NA NA NA NA NA NA NA NA NA ...
## $ LvlPressure_psi_9 : num [1:272972] NA NA NA NA NA NA NA NA NA NA ...
## $ LvlTemp_C_9 : num [1:272972] NA NA NA NA NA NA NA NA NA NA ...
## $ LvlDepth_m_9 : num [1:272972] NA NA NA NA NA NA NA NA NA NA ...
## $ RECORD : num [1:272972] 0 1 2 3 4 5 6 7 8 9 ...
## $ CR6Battery_V : num [1:272972] 13.3 13.2 13.2 13.2 13.3 ...
## $ CR6Panel_Temp_C : num [1:272972] 33.4 34 34.6 35.2 35.6 ...
## $ Flag_ThermistorTemp_C_surface: num [1:272972] 0 0 0 0 0 0 0 0 0 0 ...
## $ Flag_ThermistorTemp_C_1 : num [1:272972] 0 0 0 0 0 0 0 0 0 0 ...
## $ Flag_ThermistorTemp_C_2 : num [1:272972] 0 0 0 0 0 0 0 0 0 0 ...
## $ Flag_ThermistorTemp_C_3 : num [1:272972] 0 0 0 0 0 0 0 0 0 0 ...
## $ Flag_ThermistorTemp_C_4 : num [1:272972] 0 0 0 0 0 0 0 0 0 0 ...
## $ Flag_ThermistorTemp_C_5 : num [1:272972] 0 0 0 0 0 0 0 0 0 0 ...
## $ Flag_ThermistorTemp_C_6 : num [1:272972] 0 0 0 0 0 0 0 0 0 0 ...
## $ Flag_ThermistorTemp_C_7 : num [1:272972] 0 0 0 0 0 0 0 0 0 0 ...
## $ Flag_ThermistorTemp_C_8 : num [1:272972] 0 0 0 0 0 0 0 0 0 0 ...
## $ Flag_ThermistorTemp_C_9 : num [1:272972] 0 0 0 0 0 0 0 0 0 0 ...
## $ Flag_RDO_mgL_5 : num [1:272972] 0 0 0 0 0 0 0 0 0 0 ...
## $ Flag_RDOsat_percent_5 : num [1:272972] 0 0 0 0 0 0 0 0 0 0 ...
## $ Flag_RDOTemp_C_5 : num [1:272972] 0 0 0 0 0 0 0 0 0 0 ...
## $ Flag_RDO_mgL_9 : num [1:272972] 2 2 2 2 2 2 2 2 2 2 ...
## $ Flag_RDOsat_percent_9 : num [1:272972] 0 0 0 0 0 0 0 0 0 0 ...
## $ Flag_RDOTemp_C_9 : num [1:272972] 0 0 0 0 0 0 0 0 0 0 ...
## $ Flag_EXOTemp_C_1 : num [1:272972] 7 7 7 7 7 7 7 7 7 7 ...
## $ Flag_EXOCond_uScm_1 : num [1:272972] 7 7 7 7 7 7 7 7 7 7 ...
## $ Flag_EXOSpCond_uScm_1 : num [1:272972] 7 7 7 7 7 7 7 7 7 7 ...
## $ Flag_EXOTDS_mgL_1 : num [1:272972] 7 7 7 7 7 7 7 7 7 7 ...
## $ Flag_EXODOsat_percent_1 : num [1:272972] 7 7 7 7 7 7 7 7 7 7 ...
## $ Flag_EXODO_mgL_1 : num [1:272972] 7 7 7 7 7 7 7 7 7 7 ...
## $ Flag_EXOChla_RFU_1 : num [1:272972] 7 7 7 7 7 7 7 7 7 7 ...
## $ Flag_EXOChla_ugL_1 : num [1:272972] 7 7 7 7 7 7 7 7 7 7 ...
## $ Flag_EXOBGAPC_RFU_1 : num [1:272972] 7 7 7 7 7 7 7 7 7 7 ...
## $ Flag_EXOBGAPC_ugL_1 : num [1:272972] 7 7 7 7 7 7 7 7 7 7 ...
## $ Flag_EXOfDOM_RFU_1 : num [1:272972] 7 7 7 7 7 7 7 7 7 7 ...
## $ Flag_EXOfDOM_QSU_1 : num [1:272972] 7 7 7 7 7 7 7 7 7 7 ...
## $ Flag_EXOTurbidity_FNU_1 : num [1:272972] 7 7 7 7 7 7 7 7 7 7 ...
## $ Flag_EXOPressure_psi : num [1:272972] 7 7 7 7 7 7 7 7 7 7 ...
## $ Flag_EXODepth_m : num [1:272972] 7 7 7 7 7 7 7 7 7 7 ...
## $ Flag_EXOBattery_V : num [1:272972] 7 7 7 7 7 7 7 7 7 7 ...
## $ Flag_EXOCablepower_V : num [1:272972] 7 7 7 7 7 7 7 7 7 7 ...
## $ Flag_EXOWiper_V : num [1:272972] 7 7 7 7 7 7 7 7 7 7 ...
## $ Flag_LvlPressure_psi_9 : num [1:272972] 7 7 7 7 7 7 7 7 7 7 ...
## $ Flag_LvlTemp_C_9 : num [1:272972] 7 7 7 7 7 7 7 7 7 7 ...
Let’s look at the flag Frequency for each variable. As a reminder here are the flag codes Flag values
0: no flag
1: value removed due to maintenance and set to NA
2: negative or outlier value removed and set to NA, see Methods section for more detail on QAQC process
3: negative values set to 0
4: value removed due to fouling and set to NA
5: questionable value due to potential fouling
6: very questionable value due to potential fouling. Values adjusted using a linear or square root function to match high-resolution CTD profiles are given in RDO_mgL_5 and RDO_sat_percent_5
7: missing data
8: Value corrected using a constant offset due to two thermistor malfunctions in Fall 2020
## Flag_ThermistorTemp_C_surface
## 0 1 5 7 <NA>
## 269512 2013 1 1446 0
## Flag_ThermistorTemp_C_1
## 0 1 5 7 8 <NA>
## 252526 4472 1 1446 14527 0
## Flag_ThermistorTemp_C_2
## 0 1 5 7 <NA>
## 269361 2164 1 1446 0
## Flag_ThermistorTemp_C_3
## 0 1 5 7 <NA>
## 269620 1905 1 1446 0
## Flag_ThermistorTemp_C_4
## 0 1 5 7 8 <NA>
## 252622 4472 1 1446 14431 0
## Flag_ThermistorTemp_C_5
## 0 1 5 7 <NA>
## 268908 2617 1 1446 0
## Flag_ThermistorTemp_C_6
## 0 1 5 7 <NA>
## 268908 2617 1 1446 0
## Flag_ThermistorTemp_C_7
## 0 1 5 7 <NA>
## 268908 2617 1 1446 0
## Flag_ThermistorTemp_C_8
## 0 1 5 7 <NA>
## 268908 2617 1 1446 0
## Flag_ThermistorTemp_C_9
## 0 1 5 7 <NA>
## 268908 2617 1 1446 0
## Flag_RDO_mgL_5
## 0 1 2 3 5 6 7 <NA>
## 216159 3624 14 37437 12950 220 2568 0
## Flag_RDOsat_percent_5
## 0 1 2 3 5 6 7 <NA>
## 216147 3624 26 37437 12950 220 2568 0
## Flag_RDOTemp_C_5
## 0 1 2 5 7 <NA>
## 268038 2335 12 1 2586 0
## Flag_RDO_mgL_9
## 0 1 2 3 5 6 7 <NA>
## 193906 4082 6911 51428 12303 3214 1128 0
## Flag_RDOsat_percent_9
## 0 1 2 3 5 6 7 <NA>
## 198241 4082 293 53710 12303 3214 1129 0
## Flag_RDOTemp_C_9
## 0 1 5 7 <NA>
## 269204 2617 1 1150 0
## Flag_EXOTemp_C_1
## 0 1 2 5 7 <NA>
## 258543 5441 13 1 8974 0
## Flag_EXOCond_uScm_1
## 0 1 2 5 7 <NA>
## 255166 5420 3388 44 8954 0
## Flag_EXOSpCond_uScm_1
## 0 1 2 5 7 <NA>
## 255165 5420 3389 44 8954 0
## Flag_EXOTDS_mgL_1
## 0 1 2 5 7 <NA>
## 255165 5420 3389 44 8954 0
## Flag_EXODOsat_percent_1
## 0 1 2 3 5 7 <NA>
## 256282 6718 19 1028 1 8924 0
## Flag_EXODO_mgL_1
## 0 1 2 3 5 7 <NA>
## 254157 6718 19 1021 1 11056 0
## Flag_EXOChla_RFU_1
## 0 1 2 3 4 5 7 <NA>
## 256229 5441 175 7 6 2140 8974 0
## Flag_EXOChla_ugL_1
## 0 1 2 4 5 7 <NA>
## 256322 5441 166 5 2064 8974 0
## Flag_EXOBGAPC_RFU_1
## 0 1 2 3 5 7 <NA>
## 255752 5441 17 672 2116 8974 0
## Flag_EXOBGAPC_ugL_1
## 0 1 2 3 5 7 <NA>
## 252992 5441 18 3581 1966 8974 0
## Flag_EXOfDOM_RFU_1
## 0 1 2 3 5 7 <NA>
## 256999 6882 117 19 1 8954 0
## Flag_EXOfDOM_QSU_1
## 0 1 2 3 5 7 <NA>
## 256998 6882 118 19 1 8954 0
## Flag_EXOTurbidity_FNU_1
## 0 1 2 3 5 7 <NA>
## 57611 17768 1127 24 1 196441 0
## Flag_EXOPressure_psi
## 0 1 2 5 7 <NA>
## 258548 5441 8 1 8974 0
## Flag_EXODepth_m
## 0 1 2 5 7 <NA>
## 258548 5441 8 1 8974 0
## Flag_EXOBattery_V
## 0 1 2 5 7 <NA>
## 258555 5441 1 1 8974 0
## Flag_EXOCablepower_V
## 0 1 2 5 7 <NA>
## 258539 5441 17 1 8974 0
## Flag_EXOWiper_V
## 0 1 2 5 7 <NA>
## 258124 5441 432 1 8974 0
## Flag_LvlPressure_psi_9
## 0 1 5 7 <NA>
## 165746 2617 1 104608 0
## Flag_LvlTemp_C_9
## 0 1 5 7 <NA>
## 165746 2617 1 104608 0